SYNCHRONOUS VIDEO AND DATA ANNOTATIONS 



Technical Field of the Invention 

The present invention relates to synchronous 
annotation of data to video, such as surveillance video 
taken of a secured area. 

Background of the Invention 

Video surveillance systems are currently used 
to video record areas requiring protection from 
intruders, unauthorized use, criminal activities, etc. 
When an improper activity occurs in the protected area, 
investigators can view the video recording of the 
protected area in order to develop evidence useful in 
detecting and/or successfully prosecuting those who 
engaged in the improper activity. 

However, finding a segment of the surveillance 
video that relates to the improper activity is laborious 
and time consuming. The video recordings acquired by 
current video surveillance systems contain only images 
captured by video cameras. Therefore, all or a 
substantial part of the surveillance video must be viewed 
in order to find the relevant segment. Current video 
surveillance systems do not also record or otherwise 
capture data annotations from other sources of 
information that would make it easier to find relevant 
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video segments. Hence, when current video recordings are 
searched during an investigation, current video search 
methods rely only on temporal data and visual cognition 
of the viewer, which makes searches laborious, time 
5 consuming, and sometimes ineffective. 

The present invention involves annotating video 
recordings with data from other sources. Such data 
annotations are synchronized or linked to the video 
recordings and can enable a user to more easily locate 
10 video segments of interest. 



Summary of the Investigation 

In accordance with one aspect of the present 
invention, a surveillance system comprises a camera, an 

15 input device, and a server. The camera is arranged to 
output images of a protected area. The input device is 
arranged to provide a data annotation. The server is 
arranged to synchronously store the images and the data 
annotation so that the data annotation can be used to 

20 search for a segment of the images. 

In accordance with another aspect of the 
present invention, a method comprises the following: 
storing surveillance video in a memory; storing data 
annotations in the memory, wherein the data annotations 
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are useful in searching for a video segment of the 
surveillance video of interest; and, synchronizing the 
stored data annotations to the corresponding video 
segments of the stored video so that the data annotations 
5 can be used to search for the video segment of interest. 

In accordance with still another aspect of the 
present invention, a surveillance method comprises the 
following: capturing images of a protected area; 
storing the images in a computer readable memory; 

10 storing data annotations in the computer readable memory, 
wherein the data annotations are searchable using a 
search criteria; and, storing a link that links the 
stored data annotations to corresponding image segments 
of the stored images so that the data annotations can be 

15 used to search for an image segment of interest. 



Brief Description of the Drawings 

The features and advantages of the present 
invention will become more apparent from a detailed 
20 consideration of the invention taken in conjunction with 
the drawings in which: 

Figure 1 illustrates a video/data surveillance 
system according to an embodiment of the present 
invention; 
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Figure 2 illustrates a flow chart for a data 
tag recording program executed by the video server of the 
video/data surveillance system shown in Figure 1; 

Figure 3 illustrates a flow chart for an SQL 
5 data manipulation program executed by the video server of 
the video/data surveillance system shown in Figure 1; 

Figure 4 illustrates a flow chart for a data 
searching program executed by the video server of the 
video/data surveillance system shown in Figure 1; and, 
10 Figure 5 illustrates a flow chart for a 

playback program executed by the video server of the 
video/data surveillance system shown in Figure 1. 



Detailed Description 

15 Figure 1 illustrates a video/data surveillance 

system 10 that includes one or more video cameras such as 
video cameras 12 and 14 that are positioned so that they 
can capture video of one or more protected areas. 
Cameras other than video cameras can be used for the 

20 video cameras 12 and 14. For example, thermal imagers, 
IR cameras, etc. could be used. Moreover, the video 
cameras 12 and 14 may comprise any mix of video cameras, 
thermal imagers, IR cameras, etc. such as a video camera 
and a thermal imager, an IR camera and a thermal imager, 
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a video camera and an IR camera, etc. The video cameras 
12 and 14 feed video to corresponding camera streamers 16 
and 18. The camera streamers 16 and 18 compress and 
otherwise format the video from the video cameras 12 and 
5 14 for efficient storing and processing by a video server 
20 coupled to the camera streamers 16 and 18. Such 
compression and formatting saves bandwidth and storage 
space . 

The video server 20 stores the video streams on 
10 a suitable storage medium, and permits the stored video 
recordings to be viewed (as shown in Figure 1) and 
searched by an investigator or other user during an 
investigation or otherwise. The camera streamers 16 and 
18 and the video server 20 may be camera streamers 
15 provided by Axis Communications, and the video server 20 
may be a standard desktop server provided by Dell. 
However, it should be understood that other camera 
streamers and/or video servers can be used instead. 
Also, the functions of the camera streamers 16 and 18 and 
20 of the video server 20 can be combined into a single 
device or divided between multiple devices. 

The video/data surveillance system 10 also 
includes an input device 22 such as a barcode scanner, a 
keyboard having one or more keys, an audio device that 
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may or may not convert voice to text, etc. If a keyboard 
is used as the input device 22 , a QWERTY keyboard 
typically used with typewriters or computers may be used. 
One or more such devices can be used as the input device 
22. The input device 22 can be hardwired to the video 
server 20, or the input device 22 may include a 
transmitter to transmit data annotations to the video 
server 20 by way of RF, infrared, sonic, or other 
signals. The input device 22 can additionally or 
alternatively by a device that provides biometrics 
signatures such as from fingerprinting, facial 
recognition, retina scans, etc. 

The input device 22 may be used by a person 
such as a security guard or a foot patrol to annotate the 
video being recorded and stored in the video server 20. 
For example, such a person may enter data indicating that 
conditions are normal, or the person may enter data 
indicating an abnormality such as an explosion, a 
robbery, a broken window or door, an intruder, a 
suspicious activity, etc. The video server 20 includes 
software components that cause the data input by the 
input device 20 to be synchronously annotated to the 
video being recorded from the cameras 12 and 14. The 
annotated data is useful in facilitating a search for a 
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video segment of interest. For example, the annotated 
data can help an investigator to locate video segments 
that recorded abnormalities. 

The video server 20 is programmed with software 
5 components so that data annotations, which may be 

referred to alternatively as data tags and which are 
entered by use of the input device 22, can be stored in 
the memory of the video server 22 in a format that 
permits the data tags to be searched. For example, the 

10 data tags may be stored in SQL readable form. Thus the 
software components permit the data tags to be searched 
so that the annotated video segments can be more easily 
located and played back to a user. The memory of the 
video server 22 can be disk storage, RAM, EPROM, etc. 

15 These software components includes a data tag 

recording program 30 shown by way of a flow chart in 
Figure 2. A block 32 of the data tag recording program 
30 receives the data tag input from the input device 22. 
A block 34 of the data tag recording program 30 then time 

20 stamps the received data tag with the time that the video 
server 20 receives the data from the input device 22 and 
saves both the data tag and its time stamp in memory of 
the video server 20. This time stamp synchronizes the 
data tag with the corresponding (i.e., annotated) video 
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segment of the video also saved in the memory of the 
video server 20. Preferably, but not necessarily, the 
time stamp of the data tag and the video count associated 
with the video recording are in the same format. 

When the data tag is stored in memory of the 
video server 20, an SQL program 40 is executed as shown 
by the flow chart of Figure 3 so as to convert the data 
tag into SQL readable form. Accordingly, a block 42 of 
the SQL program 40 receives the data tag to be stored, 
and a block 44 displays the data tag in an input text 
box. This input text box may be a standard text input 
box that is used by Internet search engines to input text 
to be searched. However, the input text box may take 
other forms. A block 4 6 also embeds the data tag into an 
SQL insert statement. To then save the data tag into a 
database searchable in SQL, a block 48 executes the SQL 
insert statement by opening a session, executing the 
insert statement, and closing the session. 

When a user wishes to find and view a 
particular segment of video, a data searching program 50 
shown by way of a flow chart in Figure 4 is executed by 
the video server 20. A block 52 receives an input data 
tag search string drafted by the user. This search 
string contains keywords or other search criteria useful 
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in finding a relevant data tag. A block 54 automatically 
composes an SQL query based on the data tag search 
string. The search using this SQL query returns all rows 
of the SQL database containing the search string. These 
5 rows include the data annotations found as a result of 

the search. A block 56 formats the returned row(s) so as 
to display on the monitor of the video server 20 the 
formatted returned row(s) alongside the corresponding 
video. This display thus includes the formatted returned 

10 row(s) and the video count that corresponds to the data 
tag and that designates the annotated video segment. 
Alternatively or additionally , it is also possible to 
view the video directly from the time given in the time- 
stamp of the tagged data. A block 58 permits the user to 

15 select the video sequences for playback that match all or 
some of the tagged string by use of the graphical user 
interface . 

A playback program 60 shown by way of a flow 
chart in Figure 5 is executed by the video server 20 so 
20 as to play back the selected video. A block 62 begins 
the playback of the selected video for display on the 
monitor of the video server 20. A block 64 determines 
whether the current playback time counter has data tags. 
The playback time counter may be simply one or more 



-9- 



Attorney Docket 
H0005960 SBE 0760 



times. A playback time counter having data tags means 
that there is at least a row of data linking time and 
data tags. The decision made by the block 64 is to check 
if, at the current playback time, there is/are 
5 corresponding data tag(s) stored in memory. That is, the 
block 64 determines whether the time indicated by the 
current playback time counter associated with the video 
matches the time stamps of the data tags. If the current 
playback time counter has data tags, then a block 66 

10 causes the data tags to be also displayed on the monitor 
of the video server 20. If the current playback time 
counter does not have data tags, a block 68 determines 
whether the user has terminated the playback. If the 
block 68 determines that the user has not terminated the 

15 playback, playback of the video segments continues. On 
the other hand, if the block 68 determines that the user 
has terminated the playback, playback is stopped. 

A digital video management system providing an 
exemplary environment for the present invention is 

20 disclosed in WO 01/13637 Al . 

Certain modifications of the present invention 
have been discussed above. Other modifications will 
occur to those practicing in the art of the present 
invention. For example, video and data tags are 
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displayed on the monitor of the video server 20. 
However, video and/or data tags can be displayed by use 
of devices other than the monitor of the video server 20. 
For example, the video server 20 can send the video 
5 and/or data tags to other monitors, to projection 
screens, etc. 

Also, as disclosed above, a time stamp is used 
to link a data tag with the video that the data tag 
annotates. Accordingly, the data tags can be searched 

10 for a specific data tag to more easily find a video 

segment of interest that is linked to that specific data 
tag, and the time stamps of data tags found from the 
search can be compared to the video count of the video to 
identify the video segment of interest. However, links 

15 other than data tags can be used to associate data tags 

with their corresponding segments of video. For example, 
flags or pointers can be used to link data tags to their 
corresponding video segments. Also, links such as 
temperature stamps or luminosity stamps could be used. 

20 For temperature stamps, for example, thermal video of an 
eguipment that has exceeded certain temperatures could be 
retrieved. For luminosity stamps, for example, video 
images of a certain lighting conditions could be 
retrieved . 
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Accordingly, the description of the present 
invention is to be construed as illustrative only and is 
for the purpose of teaching those skilled in the art the 
best mode of carrying out the invention. The details may 
5 be varied substantially without departing from the spirit 
of the invention, and the exclusive use of all 
modifications which are within the scope of the appended 
claims is reserved. 
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